Repository navigation
Conversation
mlieberman85
left a comment
There was a problem hiding this comment.
Nice start. A few gaps before this fully closes #542:
- Remediation reads without
expected_level, but on a miss it re-audits at level 3, so a level-1 cache is still a hit. Please passexpected_level=3. --include/--excludeandprofile=also produce filtered audits that currently look unfiltered. Please record the full scope (afilteredflag or the control-ID set).- A filtered audit still replaces an unfiltered cache, and the remediation read ignores
[stores.cache]. Fix both here, or change "Fixes #542" to "Refs". - Bump
CACHE_VERSION, or treat a missingtagskey as a miss, so old cache files aren't read as unfiltered. - Spec-first: describe the envelope change in
framework-design.md10.4. - Add the model version to
Assisted-byand name the tool in the PR body.
Drafted with Claude Code; reviewed and posted by me.
16fa6e3 to
56b39cc
Compare
|
Addressed.
Tests: 38 in Three failures in the long I also left |
mlieberman85
left a comment
There was a problem hiding this comment.
Thanks, this covers everything I asked. Two things before it can merge:
- Rebase onto main. It conflicts in
orchestrator.py,cli.pyandaudit.py, so CI hasn't run. #552 changed_load_merged_storesto(framework_name, operator). After the rebase,_load_merged_stores(local_path, "openssf-baseline")would raise inside thetry, and every remediation would quietly re-run the audit. Please resolve the operator configuration the way_run_baseline_checksdoes. Ideally, also move the cache-key derivation into one helper that bothtools/audit.pyand the orchestrator call, so the two can't drift apart again. - Profile-filtered audits.
audit_openssf_baseline(profile=...)narrowscontrolsitself and passestags=None, so its cache entry still saysfiltered=False, and remediation reads it as a full audit. Please fail closed: when a caller passescontrolswithout sayingcache_filtered, treat the audit as filtered. Callers that pass the full set (the MCP tool without a profile, the CLI without filters) then passcache_filtered=Falseexplicitly. Please add a test for the profile case.
On the three tests that fail in a full-suite run: main now passes the full suite (5210 passed after #548 made the gh calls offline). If they still fail after the rebase, please open an issue with the output.
Drafted with Claude Code; reviewed and posted by me.
56b39cc to
913771a
Compare
|
Rebuilt on current main and force-pushed (one commit). Main had rewritten the three conflicting files, so I re-applied the change instead of resolving the conflicts in place.
Checked through the CLI on a committed one-file repository:
One existing test changed: Full suite: 5167 passed, 19 skipped. The three tests that failed for me before did not fail. Note for #576: it wires Drafted with Claude; reviewed and posted by me. |
mlieberman85
left a comment
There was a problem hiding this comment.
Thanks, the rebuild covers both round-2 asks, and the W-03 probe now re-audits instead of saying "all controls are passing". A few things before merge:
- Remediation now reads through
[stores.cache]but still invalidates the default store (orchestrator.py:1289,darnit_baseline/tools.py:512). With a configured store and a dirty tree,create_security_policywrites SECURITY.md and the next remediation still reads the old OSPS-VM-02.01 FAIL. Please invalidate through the same store/key as the read (a helper besideread_full_audit_cache), and update the invalidation requirement in framework-design.md 10.4 (line ~1899). - Please refresh the PR body: it still says "partially" and lists store routing and filtered-run overwrite as not included, and it should name the AI tool used.
cli.py:341lists the filter flags by hand, so #576's--profilewould be cached as unfiltered. Could the CLI derivecache_filteredfrom whether the filtered control list is smaller than the loaded one? That survives future flags.- Smaller: nothing reads
audit-cache-filtered, so not writing filtered audits at all would be simpler, andtags/expected_tagswould then be redundant withfiltered. The profile test could run the realaudit_openssf_baseline(profile=...)and assertread_full_audit_cache(...) is Noneinstead of patchingrun_sieve_audit.
Drafted with Claude Code; reviewed and posted by me.
A filtered audit left a cache that remediation read as a full audit. On main, `darnit audit --tags level=1` writes 26 results and remediate_audit_findings then acts on those as if all 66 controls had run. The cache envelope now records the audit's scope: framework, level, tag filter, and whether any filter narrowed the control set. CACHE_VERSION is 2 and must match exactly, so older envelopes, which cannot describe their own scope, are ignored and rebuilt. A filtered audit is written under its own key, so it never replaces the full-audit entry. run_sieve_audit records an audit as filtered when it has tags, when the caller says so, or when the caller passed its own `controls` without saying. That last case fails closed: the function cannot tell a narrowed list from the full set. Callers with the full set say cache_filtered=False (the generic MCP audit tool); the CLI and the Baseline tool compute it from --tags/--include/--exclude and from the profile. Remediation reads through read_full_audit_cache, which resolves the operator configuration and the store the way run_sieve_audit does, derives the key with the same helper (audit_cache_key), and accepts only an unfiltered level 3 audit of its framework. Rebuilt on current main rather than rebased: main had rewritten the three conflicting files since the original commit. Drafted with Claude, edited by me. Signed-off-by: Marc-cn <130138935+Marc-cn@users.noreply.github.com> Assisted-by: Claude:claude-fable-5-1
…igured store Addresses the review of Oct 12 on this PR. - A filtered audit is not written to the cache at all. That replaces the separate "-filtered" key and the tags/filtered envelope fields, which nothing read. The read checks framework and level only. - invalidate_full_audit_cache() resolves the same store and key as read_full_audit_cache(). Remediation used the default store, so with a configured [stores.cache] the old audit stayed in place. - The CLI derives cache_filtered from the control count: fewer controls than were loaded means filtered. That covers --profile (#576) and any future flag without listing them. - The profile test runs the real audit_openssf_baseline() and checks that a profile run leaves no full-audit cache. - Spec 10.4 updated for the invalidation and the envelope. Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Marc-cn <130138935+Marc-cn@users.noreply.github.com>
913771a to
732c143
Compare
|
@mlieberman85 the Oct 12 points, rebased onto main:
5173 tests pass. |
Summary
darnit audit --tags level=1wrote a cache that remediation then read as a full audit (#542). Two rules fix it:--tags,--include,--excludeor a profile does not write the audit cache. The CLI decides "filtered" by comparing the control count after filtering with the count loaded, so any future flag is covered.read_full_audit_cache()andinvalidate_full_audit_cache()share the store resolution, so an operator's[stores.cache]is honoured on both sides. Before, invalidation always hit the default store and a configured store kept the stale audit.The cache envelope records
frameworkandlevel;CACHE_VERSIONis 2, so caches from earlier versions are ignored and rebuilt.Type of Change
Framework Changes Checklist
docs/architecture/framework-design.md) if behavior changed (section 10.4)uv run python scripts/validate_sync.py --verboseand it passesTesting
Tests pass locally (
uv run pytest tests/ -v): 5173 passed, 19 skippedAdded tests for new functionality (if applicable)
Linting passes (
uv run ruff check .)tests/darnit/core/test_audit_cache.py: the filtered decision, one key per repository, read hit and miss cases, invalidation through the default store, and invalidation through a configuredlocal-fsstore. The last one shows the oldinvalidate_audit_cache(path)leaving the configured entry in place and the new helper expiring it.tests/darnit_baseline/test_audit_cache_scope.py: runs the realaudit_openssf_baseline()with and without a profile, and checksread_full_audit_cache()is None only for the profile run.By hand, on a temp repository:
audit -t level=1MISS,audit --include XMISS, full audit HIT (66 results),audit -t level=1after the full audit still HIT with 66.AI assistance
Claude Fable 5.1 wrote the changes and drafted this description. I ran every command and checked the results myself. The commits carry an
Assisted-by: Claude:claude-fable-5-1trailer.Additional Notes
darnit audit --profile(#576) is covered by the control-count rule here, so #576 needs no cache code of its own.